(12) 



UK Patent Application , 19 ,GB ,,,,2 318012 „ 3 ,A 



(43) Date of A Publication 08.04.1998 



(21) Application No 9720896.1 

(22) Date of Filing 02.10.1997 



(30) Priority Data 
(31) 9620838 



(32) 02.10.1996 (33) GB 



(71) Applicant(s) 

GEC-Marconi Limited 

(Incorporated in the United Kingdom) 

The Grove, Warren Lane, STANMORE, Middlesex, 
HA7 4LY, United Kingdom 

(72) Inventor(s) 

Simon Howard Spencer 

(74) Agent and/or Address for Service 
J Waters 

GEC Patent Department, Waterhouse Lane, 
CHELMSFORD, Essex, CM1 2QX, United Kingdom 



(51) INT CL 6 

H04N 9/07 

(52) UK CL (Edition P) 

H4F FCB FD1A1 FD181 FD1B3 FD1B9 FD1D9 FD1P9 
FD1G4 FD1Q7 FD1V FD12X FD18R FD25L FD26G 
FD27C1 FD27C9 FD27H FD27L FD27M FD27T1 FD30E 
FD30G FD30H FD30K FD30P FD53D FD54 FD83B FHG 

(56) Documents CHed 
GB 2135853 A 

(58) Field of Search 

UK CL (Edition 0 ) H4F FCB FDF FEG FGB FHG 

INT CL 6 H04N 9/00 9/04 9/07 9/077 9/083 17/00 17/02 

Online:WPI 



(54) Video cameras 

(57) A camera producing colour video outputs may employ an image intensifier 5 and a monochrome CCD 
sensor 3. The scene is exposed through filters F3, F 4 which each contain stripes of subtractive primary colours 
alternating with clear regions. The video is decoded using waveforms derived from a memory 15 which 
records the output of the sensor 3 when imaged on, for example, primary colours such as red and blue, 
producing trains of pulses which identify the spatial regions of each line corresponding to the spatial areas of 
the pattern on the sensor produced by the colour filters F3, F 4 , in order that actual scenes can be decoded 
using the stored waveforms, for example using sample and hold devices. A colour interference signal is 
generated in matrix addition circuitry to produce a luminance signal which has stripe visibility removed from 
it, permitting high resolution video signals to be obtained. 
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At least one drawing originally filed was informal and the print reproduced here is taken from a later filed formal copy. 
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# # 2318012 

1 

CAMERAS 

This invention relates to cameras. 

The invention is particularly applicable to cameras which produce colour video outputs 
(including false colour) such as may be fed to a colour display. 

5 

The invention is also particularly applicable to such cameras in which a single radiation- 
sensitive sensor is used to produce the colour video outputs. Typically the sensor is 
scanned in a raster or sampled as in a solid state array. 

10 Various proposals have been made to solve the problem of how to reproduce colours 
using a single sensor. One of the earliest proposals was Baird's system of aiming a 
monochrome television camera through a rotating filter, successive segments of which 
passed red, bhie and green light. The receiver used a similar rotating filter synchronised 
with the one at the camera. 

15 

US-A-2 733 291 and US-A-3 378 633 disclose the concept of exposing monochrome 
light-sensitive devices (a vidicon in the case of the former and cine film in the case of the 
latter) to produce colour television outputs, each exposing the light-sensitive device 
through a filter consisting of stripes of two colours which repeat in the scanning direction 
20 at different frequencies. A tuned circuit is used to select the respective colour 
components in the output. In the latter patent, the filter consists of alternate clear and 
cyan vertical stripes, overlying alternate clear and yellow stripes inclined relative to the 
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vertical suipes This provides a grouping of four elemental colour areas which is repeated 
over the entire area of the filter and is ,maged over the entire area of each frame of the 
cine film. Where the transparent stripes overlap, the light-sensitive device produces a 
signal corresponding to the fell luminance signal. Where cyan (which passes green and 
5 blue Ugh.) overiaps yellow (which passes nd and green light), only the green component 
of to scene is imaged. Where die cyan and yellow stripes overlap the transparent stripes, 
the light-sensitive device images the green and blue, and green and red components of the 
scene, respectively. U is sufficient for the ration frequency of the yellow and cyan 
stripes in the direction in which the cine film is ultimately scanned in order to produce 
10 signals for a colour television receiver to be different (for exunple, by making one set of 
stripes vertical and the other set of dimensionally identical stripes inclined), for h to be 
possible to extiac, the components in the output of the scanned cine film by m«ns of 
tuned circuits. Sufficient information is provided for a low resolution lumtaance 
component to be produced, in addition to three components corresponding to the primary 
15 colours (red, green and blue). 

A disadvantage of the use of tuned circuits to extract the individual components in the 
output of the video waveform is that only modes, pictures are produced due ,0 cross-talk 
(cross-luma and cross-chroma). 



20 



When each cine frame is raster scanned in US-A-3 378 633, <h. grey scale value of 
successive pixels on each line corresponds to that for the respective colour components 
i.e white, yellow, green, cyan, in a fixed s«,u«nce. It has been proposed (Albert 
Macovski - Spatial-Frequency Encoding Techniques Applied to a One-Tube Colour 
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Television Camera, IEEE Transactions on Broadcasting, Vol BC-16, No. 4, December 
1970) to sample the grey scale of each pixel on each video line corresponding to the 
respective colour components, rather than to discriminate between the components using 
tuned circuits. However, geometrical errors resulting in indefinite registration between 
the filter and the cine film, and between the cine film and its scanner, would make it 
difficult to predict which colour components any group of pixels represents. 



It may be noted that this disadvantage does not apply in a consumer video camera, in 
which successive pixels along each line of the solid state imager are covered by gelatine 
filters which are yellow, cyan, green or transparent, because the filters are physically 
secured to the imager in this case. This means that, say, the first pixel on line 1 represents 
the scene imaged through a yellow filter, and accordingly outputs of the imager 
corresponding to the respective colour components can be derived with certainty. 



15 However, such an arrangement would not be possible if the sensor included an image 
intensifier. In this case the filters would have to be positioned in front of the image 
intensifier rather than in front of the solid state sensor, since the output of the image 
intensifier is itself monochrome. 



20 The geometrical distortion produced by an image intensifier is such that it would be 
impossible to predict the colour sequence for each line of the solid state imager based 
upon the sequence of elemental colour areas in the filter. 

In the case of a single-tube colour television image pick-up apparatus exposed through 
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c„,our strip. filter, it has been proposed .0 compete for non-Unearit, in the tub,s 
^ecUonsys.emby.heuseofa.ame^morvCGB-AO 135853). 
stores vtf» s^»s corresponding to —on of the rube with primary colours. These 
sored signals are used to contrite for non-linearities u, the image piclc-up mode of the 
^ However, the video sgnals, whiie corrected for non-linearities caused e.g. by drift, 
are nevertheless low resolution. 

It been propose* <W - A New Versatile Fun Resohmon Single-Tube Coiour 
TV r^era System", M Koub* IEEE Transactions on Broadcasting Vol BC-22 No 3 
September 1976, pp 30-35) to produce a high resolution single-tube camera. This is done 
by deriving a luminance signal which occupies a subaantial proportion of the vdeo 
bandwidth. A singled camera is exposed through a colour sripe alter, and separa* 
^outpuu^produced, ash.GB-A-2 135 853. The output of the single-tube is such 
. .uminance signal, but the problem is that the stripe pattern is superimposed, and drastic 
ffltering ha, been used in the pas, to remove the effect of the saipes Because GB-A-2 
,„ 853 uses vertical snipes, the respective colour information appears in the vdeo 
bandwidth in the form of harmonics of tire line frequency (which contains luminance 
formation, and hence there is no way of separating the chrominance information and the 
^ information necessitating severe altering of the bandwidth to produce a 

^signal. ^^.^^^^*>*>* M 
^fonnation is heaved v,th me harmonics of the Une agency in the video bandwidth, 
^ a comb filter is used to separate the chrominance and .uminance ^formation. Thts 

a bandwidth commensurate with the video bandwidth. However, the system relies on 
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using a linear scan camera tube (i.e. good geometry). If the stripe pattern on the target 
became distorted, the bandwidth of the chrominance signals would increase and would no 
longer interleave the line harmonics. 

5 It is an aim of the invention to permit high resolution colour video signals to be obtained 
using a single sensor, even if accompanied by an image intensifier, which would suffer 
from significant geometric distortion. 

The invention provides a camera comprising a sensor for receiving radiation forming an 
10 image of a scene, filter means positioned in the path of radiation incident on the sensor, 
the filter means being arranged to pass different spectral regions in different spatial 
regions, so that different spatial regions of the sensor are exposed to radiation of different 
spectral regions, decoding means for producing separate outputs from the sensor 
corresponding to the different spectral regions, the decoding means being arranged to use 
15 stored signals derived from the sensor output corresponding to exposure of the sensor 
through the filter means by radiation of reference spectral regions, and a circuit for using 
the outputs corresponding to the different spectral regions to remove visibility of the 
different spectral regions from the sensor output, to permit high resolution to be attained. 

20 In the case of visible radiation, the colour outputs are used to remove the filter pattern 
from the sensor output, thereby providing an improved bandwidth luminance signal. The 
storage of signals derived from the sensor output when illuminated by radiation of 
reference spectral regions enables the sensor output for an actual scene to be decoded 
accurately and avoids the need to predict which spatial areas of the sensor correspond to 
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which spectral regions based upon geometrical considerations as well as avoiding the need 

for tuned circuits. 

The colour output must of course be reasonably free from luminance cross-talk One 
5 way of achieving this is to use a stripe Hter the lines of which are orientated obliquely to 

the lines of the image. 

Of course, the invention is not restricted to visible radiation and extends, for example, to 
I-R radiation for thermal imaging as well. In the latter case, the colour components fed 
10 to the display could be for the same colour components as for a video camera, but could 
be arranged to correspond, by the use of suitable spectral niters, to particular infra-red 
frequency bands. In this case, the image would be a false colour image, but having two 
or more images corresponding ioM^M^*W^^^~^ 
easier to identify if displayed as false colours. 

15 

Advantageously, the camera includes a waveform generator for generating pulses derived 

derived from the sensor output. One sample and hold means may be provided for each 
spectral region. T*e stored signal may be square waveforms derived from the sensor 
20 output, for example, programmed into a memory used by the waveform generator. 

The sensor may include a CCD array, each spatial region which corresponds to a different 
spectral region including at least one, preferably at least four pixels. 




/ 



In the case of a stripe filter, for visible radiation, there could be two sets of stripes, for 
example, yellow alternating with clear inclined at an angle to one side of vertical, with the 
other set, for example, cyan alternating with clear, inclined at the same or different angle 
to the other side of vertical. The invention is particularly applicable to a sensor which 
5 includes an image intensifies for example, in order to provide a genuine colour image of 
a night-time scene. However, the invention is also applicable to sensors which respond 
in the infra-red or ultra-violet, in which case the camera would generate a false colour 
output. 

10 A camera constructed in accordance with the invention will now be described, by way of 
example, with reference to the accompanying drawings, in which: 

Figure 1 is a schematic view of the camera; 

1 5 Figure 2 is a block circuit diagram of the camera; 

Figure 3 is a block circuit diagram of the Colour Decode and Colour Encode of Figure 2; 

Figure 4a represents the waveform of the sensor output for a part of one line when 
20 imaging a blue scene; 

Figure 4b shows Waveform A, a square wave waveform derived from that of Figure 4a; 

Figure 4c shows the waveform of the sensor output of a part of a line when the camera 
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is illuminated by a red scene; 

Figure 4d shows Waveform B, a square wave waveform derived from the waveform of 
Figure 4c; 

5 

Figure 4e illustrates graphically how the individual colour components are obtained from 

waveforms A and B; 

Figure 4f shows a fragment of the filter projected onto a fragment of the CCD array as if 
10 no distortion were to take place; 

Figure 4g shows a fragment of the filter projected onto a fragment of the CCD array 

showing typical distortion; 

15 Figure 5 shows the interconnection of the memory and Waveform Generator in more 
detail; 

Figure 6a shows a line of video signal; 
20 Figure 6b shows one of the waveforms generated by the waveform generator; 

Figure 6c shows the output of the sample and hold when the waveform of Figure 6a is 
input and the scanning pulses of Figure 6b are applied; and 
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Figures 7 and 8 are colour bar charts discussed with reference to Figure 3. 

Referring to Figures 1 and 2, the camera produces colour video outputs. The camera has 
a lens 1 mounted in a casing 2, which contains a sensor comprising a CCD array 3 coupled 
5 by optical fibres 4 to the phosphors on the output surface of an image intensifier 5 at the 
focal plane of which filter means F 3 , F 4 is cemented (the position of the filters has been 
exploded for clarity in Figures 1 and 2), the front part of the image intensifier also 
including a fibre optic coupling. ¥ x is a correction filter which converts the spectral 
response of the sensor (i.e. the intensifier) to nominally photopic. The output of the CCD 
10 3 undergoes processing in circuit 1 1 to produce a raw video output and then subsequent 
colour processing in circuit 12 to produce colour video outputs. 

The CCD 3 and the image intensifier 5 being electro-optic devices, the spatial pattern of 

the colours of the filter means F 3 , F 4 imaged onto the CCD 3 will not change with age or 
15 temperature and remains fixed once the camera has been manufactured. The colour 

processing uses stored signals based upon the response of the sensor to reference colours. 

These signals are generated after manufacture of each camera by means of test circuit 13. 

Once the signals have been stored, the test circuit is disconnected and the camera needs 

no further adjustment. If the intensifier were to be replaced over the lifetime of the 
20 camera, the manufacturer would have to reconnect the test circuit to programme in new 

stored signals. 

The CCD 3 is a solid state device in the form of an array, a certain number of pixels to 
each line and a certain number of lines in the array. The scene is imaged in normal video 
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fashion in frames, each consisting of two consecutive interlaced fields. 

Referring to Figure 4f, a fragment of the filter means is illustrated. The filter F 3 consists 
of diagonal yellow (Ye) stripes, the width and the spacing of which are equal to each 
other. In other words, between the stripes the filter is transparent (indicated W for white 
in the drawings). The filter F 4 consists of similarly arranged cyan (Cy) stripes spaced by 
transparent stripes. When the two filters are superimposed, a repeating pattern of four 
elemental colours is formed, as in the prior arrangements noted earlier. 

If no distortions were produced by the lens 1, filters F„ F* image intensifier 5, and fibre 
optic couplings, the spatial pattern of Figure 4f would be projected onto the CCD array 
3. For the purposes of explanation, fragments of two lines n, n+1 of the CCD array 3, 
beginning at pixel k for each line, are illustrated in Figure 4£ as if the camera had 
distortion-free optics. 



For line n, the kth pixel receives light from the image which has passed through the cyan 
filter, and so does the k+1* pixel. The next pixel the k+2* partly receives imaging light 
through the cyan filter, and partly receives light through the transparent (white) filter. The 
k+3 ri and k+4* pixels receive imaging light through the transparent filter. The next pixel 
20 (k+5*) receives imaging light partly through the transparent filter and partly through the 
yellow filter, while the next two pixels receive light through the yellow filter. The k+8* 
pixel receives light through the yellow filter, partly through the cyan plus yellow (=green) 
filter, and the next two pixels only receive the light through the green filter. 
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Ignoring the pixels which receive light through the transition between two filter areas, the 
pixels of the n* line receive cyan, white, yellow and green, in a repeating sequence, as 
shown in Figure 4e. The effect of the pixels which receive light from two filter areas can 
be reduced by taking the average of two lines for each of the yellow cyan and green 
5 signals. For a hypothetical non-distorting arrangement, the sequence of colours regularly 
repeats. 

For line n+1, the sequence of colours exposed on the pixels again regularly repeats, but 
the sequence is now different. The k* , k+1* and k+2* pixels receive light from the scene 
10 exposed through a cyan filter, the next two pixels, the k+3 rt and the k+4* lie at the 
intersection of four filters, while the next four receive light through the yellow filter. The 
sequence for row n+1 is cyan, yellow, green. 

For each line, for this hypothetical situation of no distortion, the pixels of each line are 
1 5 illuminated by areas of light, the colour of which varies in a fixed repeating sequence. 

In reality, distortions are introduced by the lens, filters ¥ u F 2 , image intensifier 5, and by 
the fibre optic couplings. The pattern of the filter F 3 , F 4 actually projected onto the CCD 
array could in a typical example, be more like the pattern of Figure 4g. The kinks are a 
20 by-product of the use of optical fibres. Compared to the ideal pattern which would be 
projected by perfect optics, a real spatial region corresponding to a particular spectral 
region could be displaced by as much as 20 pixels along a line of 600 pixels. 

It will be apparent that there will still be a unique sequence of colours through which 
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successive pixels of each line are illuminated, but that the sequence will now no longer be 
regularly repeating and that there is no way of predicting what the (irregular) sequence 
for each line will be. 

5 The raw video consisting of consecutive lines of each field is decoded in the sense of 
identifying those parts of a line which correspond to the scene viewed through each one 
of the four colours by using stored signals which are representative of the raw video 
output when the regularly repeating stripes are illuminated with primary reference colours. 
Thus, referring to Figure 4a, the camera is first imaged on a uniform primary blue scene. 

10 Cyan, which is a combination of blue and green, will be transparent to the blue, whereas 
yellow, which consists of red and green light, does not pass blue light, and so the result 
will be (referring to Figure 4f) blue regions which extend diagonally from the top of.the 
drawing to the left of the drawing interspersed with similarly inclined dark regions. The 
blue stripes will result in the approximate sinusoidal variation shown for the fragment of 

15 one line in Figure 4a. Illumination by a uniform red scene will result in red stripes 
extending diagonally from the top of Figure 4f to the right interspersed with opaque 
stripes. Again a sinusoidal pattern will be produced (Figure 4c), but out of phase with that 
for Figure 4a. The raw video will be cyclic at the spatial (electrical) frequency of the 
cyan filter. 



20 



The lines shown in Figure 4a, 4c correspond to the hypothetical distortionless case for 
simplicity. With the actual pattern imaged on the CCD array, the traces Figure 4a and 4c 
will not be regular. 
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For each of the reference illuminations, the raw video from the CCD 3 is stored for a 
complete frame. What happens is that the approximately sinusoidal modulation passes 
through a slicer circuit 14 which produces waveform A from Figure 4a and waveform B 
from Figure 4c. These two waveforms are programmed into a memory 1 5 in the camera, 
5 each for a complete frame. The waveform generator 1 6 employs combinational logic on 
waveforms 4a and 4b to produce four waveforms, each containing just the pulses 
corresponding to one of the four colours i.e. yellow, green, cyan or white (Figure 4e). 
For example, logic "and" produces the pulses corresponding to white illumination, logic 
"not-or" produces an output corresponding to green illumination, and so on. Thus, the 
10 memory contains, for each line of each frame, pulse trains which indicate the precise 
points in time at which the raw video corresponds to illumination by the respective colour. 
The memory may be a preprogrammed memory i.e. PROM or EPROM (which selects 
parts of the waveform according to spatial location). This is used then to separate out the 
colour components from the raw video from an actual scene. 

15 

Referring to Figure 5, 6a to 6c, the raw video for one particular colour of the actual scene 
enters the sample and hold, which is controlled by pulses for that colour which are 
generated by the waveform generator using combinational logic from waveforms A and 
B in the memory. One line of the scene may be as shown in Figure 6a and the pulses 
20 which occur at the times when that line of the CCD are illuminated by one of the four 
colours shown in Figure 6b. The output of the sample and hold is as illustrated 
schematically in Figure 6c. 

Referring to Figure 3, the same is done for each of the four colour components, resulting 
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„ video waveforms corresponding «o Mte through the cyan. yeUow, green and 
.ransparen, areas of .he fiher means. These four waveforms, from a non-volatile memory, 
rf he produced every time the camera is operational. These are used to generate red, 

5 7 subtiact the green component from me yeUow component, producing a red component, 
to^er 6 and adder 8 subtract the green component from the cyan component to produce 
a blue component. At adder 9 the b.ue and red components are added to produce a 
magenta component. In addition circuit 10 the red, mage«a and blue components are 
suitably added togemer * produce the colour interference component Ac. The white 

1 o component need not be used. 

The reason for the addition of the colour interference component may be seen from 
Figures 7 and 8. If one imagines a colour bar char, for any particular line of the video 
where me successive bars are produced by imaging through the transparent, yellow, cyan 
and green filter areas, the overaU tatensny of the yellow, cyan and green areas is clean, 
.essthanthatof the white areas, since the yellow area has subtracted blue, the cyan area 
has subtracted red, and the green areas have subtracted biue and red. The corre* 
proportions for the red and blue are produced in cir^ 10 ,0 produce the corrected high 
resolunonhmunancesignalYatadder .0a. The low resolution luminance signal leaving 
box „ (Processing) would represent the grey sca.e intensity of the scene, bu, would have 
the aripe pattern visJMe on h, fomung a partem as in Fig 4g, bu, whh various grey levels 
for ,he cyan, yeUow and green areas. The matrix addition circuhry .0 removes the 
visibility of tins Stiioe partem from the raw video signal, producing a iuminance signal 
having a bandwidth occupying a significant proportion of the video bandwidth. 
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The matrix addition circuitry 10 is gated by the cyan, yellow and green pulses to produce 
the appropriate Ac for the appropriate part of the raw video waveform. Thus, at the times 
(Fig 6c) when the sample and holds produce an output for one particular colour, say, for 
green, Ac is added to the raw video at those instants to reflect the fact that the raw video 
5 is considerably reduced in intensity because the red and blue components have been 
stopped. The same happens at the instants at which the sample and holds for other 
colours produce outputs as shown in Fig 6c. Clearly, Ac is zero when the white pulses 
arrive at the matrix addition circuitry. The action of the sample and hold circuits convert 
the incoming modulated waveforms into continuous waveform components of cyan, 
10 yellow and green without any discontinuities (i.e. gaps). The action of the adders 6, 7 
produces the full set of primary signals, red, blue (and green), which are also continuous 
waveforms. 

It should be added that the addition performed at adders 8, 9 and inverters 6, 7 produces 
15 primary colours which do not have luminance cross-talk. This is made possible by the 
non-vertical nature of the stripes. 

The matrix addition circuitry 10 may include a clipping circuit to restrict Ac to 30% input 
white (Y) level at times when cyan is subtracted. Then if a noise spike is generated in the 
20 matrix addition circuitry 10, the noise spike is limited. Similarly, Ac can be restricted to 
10% (of Y) at times when yellow is subtracted (see Figures 7 and 8). 

The corrected luminance signal may be further improved by passage through a comb filter 
to remove residual colour energy. The luminance comb "stop bands" occur at frequency 
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iocations which are in between line harmonics These locations depend on stnpe angle 
and geometry errors. The benefit of the colour separation technique described is that 
significant suppression is achieved with poor geometry sensors. Further inclusion of a . 
comb filter enables even better suppression and reduction of noise (both random and 
5 static). Poor geometry sensors could no. use comb filters without the colour separator 
herein described. 

I, is desirable that the camera used for producing colour should have the following 
features, crysul tuning, unhy gamma. »nd good adulation a. the highest stripe spatial 
10 frequency (i.e. 50% at 3.7MHz). 

The red, green, blue and luminance signals pass from the Colour Decode 17 into Colour 
EncodeW Gamma correction is ^rfb^ 

of the luminance signal is added to each of the low resolution red, green and blue channels 
IS to produced high resolu&nour^ Theou, P u, 
of Colour Encode 18 is in three groups. The firs, group is three channels of red, green 
arc! blue video outputs together with synchronising waveforms. The second group is two 
channels of the luminance signal (Y) and the chrominance signal <C). The third group of 
channel, is a single channel which is used to encode either PAL, NTSC or SECAM 
20 Outgoing video from the decoder is T.V. standard encoded .0 suit external equipment (i.e. 
TV displays, video recording, r.f. links etc.). 

The waveform generator 16 is also responsible for generating TV pulses such as 
synchronisation pulses, blanking pulses and clamping pulses etc. 
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A typical resolution of the CCD array could be around 600 pixels by 600 lines. For a 
conventional TV line frequency, which corresponds to a line period of 64^xs and an active 
line period of 52ns, the maximum frequency which can be resolved along the lines would 
correspond to alternate black and white values i.e. 300 periods in 52|is i.e. a maximum 
5 frequency of around 6MHz. The response of the intensifier falls as the frequency 
increases and, for this reason, a typical frequency for, say, the cyan stripe is 3 MHz. This 
would correspond to two pixels for the cyan stripes, alternating with two for the 
interposed transparent stripes. Along a line of an array with pixels of 20\i by 20n, this 
would give a stripe frequency of 12Vi line pairs per mm (referenced to the CCD array) i.e. 
10 there would be l2Vi cyan stripes and HVi interposed transparent stripes per mm. 
Remembering that the stripes are advantageously inclined, say, at 22° on either side of the 
vertical centre-line of the array, this would equate to an actual line spacing of around 1 5 
line pairs per mm for both the cyan and yellow stripes. 

15 Defocus filter F 2 reduces alias disturbance of scene detail, and may be a lenticule, 
birefringent plate, Savart plate or cylindrical lens. For surveillance use (i.e. long focal 
length lenses, zoom lenses etc) this filter can be omitted. 

The shading errors of the intensifier are a fixed signature of the sensor and can thus be 
20 optically corrected. During camera manufacture, the shading errors can be measured (see 
the roll-off at the end of the line in Figure 4a), and an inverse profile pattern can be 
produced (by for example a computer line transparency printer). It is beneficial to 
perform shading correction opticaliy using filter F 5 rather than electronically as this avoids 
errors from scenes of low contrast. 
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lt w», be seen that th. proposed system uses a techntque which aUows sampling emotion 
,o derive colour waveforms. It is belteved that this approach is superior to al! other 
approaches as it enables poor performance cameras to be used. 

The innovate* are the method of deriving the sample patterns, the method of storing the 
sample patterns, the method of removmg colour errors from luminance signals (i.e. Ac 
catenation), the method of removing luma errors from chroma signaU (i e. co,our 
separation) and the method to remove shading. 

0 The technique has the following benefits, unproved huna cross-talk Le. the rejection of 
coiour signais onto the huna channel wffl be consent over the tota. area of the picture; 
improved chroma cross^ll, the rejection of colour cross-* of one colour onto another 
wiU be — over the tota! picture area (co.our shading win no, arise or "conaant 
luminance'' wu. exist); the rejection performance will not degrade whh either temperature 
„ or age (a featitre of intensify); the technique can be implemented by tetany digiul 
techniques (,.e. iow cos, smaU size and sti*.e performance); me intensified co.our pattern 
can be smaU in size, low on power and low cost; improved rejection can be achieved by 
adding farmer filtering by using one or two TV line c^mb filters; whit, highhghts in the 
^ or da* parts of a scene wUl no, be co.oured "green" i.e. the system has a desirable 
20 -fadco-gtey feature, choice of stiHped filters is not dependent upon compromising 
performances stiipe ang.es couid be ± 45' fbrNTSC syaems, ±22.5° for PAL systems 
or even between ±20' to «0-. The pitch stripe can be the same for both 

colours (cyan and yeUow). This allows the stripe frequency ,o be low (e.g. 2.5 
MHz,, thereby aKowmg the use of poor resolution modulation transfer function sensors 
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such as intensifiers; the colour beat pattern i.e. frequency difference between colour filters 
will be hardly perceptible throughout the total picture area as the sample pulses are 
synchronised to the patterns - cameras with poor geometry performance (10%) can be 
used as the colour performance is not impaired by geometry; and colour errors caused by 
5 poor corner focussing (Le. astigmatism) will be seen as lesser colour saturation rather than 
colour (hue) errors - the shading corrector (optical) lessens the saturation error. 
Examples of intensifier types for use with this technique are: first generation, second 
generation; or electron bombarded CCD. Intensifiers with a cover glass (e.g. third 
generation) can be used with a relay lens. The point here is that the focal plane lies within 
10 the casing of the intensifier making it impossible for the filter to be secured at this point. 
The relay lens puts the filter at the focus of the relay lens which is itself focuses at the 
focus of the intensifier. All sensors should have a linear transfer function i.e. unity gamma. 

The camera is not limited to use with an intensifier, and could be used on a CCD without 
15 the intensifier. Further, types of solid state sensor other than CCD could be used. 

While the description has been in relation to the use of cyan and yellow stripes, it would 
be possible to use any combination of two subtractive primary colours, or three 
subtractive primary colours could be used, possibly without the transparent regions, and 
20 primary colours themselves could be used, but this would reduce performance 
considerably. The reference illuminating colour would of course depend upon the colours 
in the filter. Other angles of the stripes and spacings could be used. Further, instead of 
having two sets of stripes inclined on each side of vertical, one set of stripes, for example, 
cyan, yellow, green and clear, or red, green and blue, inclined obliquely to the lines, could 
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be used. Also, patterns other than stripes could be used as circles, hexagons etc. 

Finally, the invention is applicable to wavelengths outside the visible range, for example 
the infra-red range where the filters F 3 and F 4 will correspond to particular infra-red 
spectral regions in the way that the yellow and cyan stripes correspond to particular visible 
regions, resulting in a false colour picture where objects emitting in three different 
frequency ranges are separately shown. 
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CLAIMS 

1 . A camera comprising a sensor for receiving radiation forming an image of a scene, 
filter means positioned in the path of radiation incident on the sensor, the filter means 
being arranged to pass different spectral regions in different spatial regions, so that 
different spatial regions of the sensor are exposed to radiation of different spectral regions, 
decoding means for producing separate outputs from the sensor corresponding to the 
different spectral regions, the decoding means being arranged to use stored signals derived 
from the sensor output corresponding to exposure of the sensor through the filter means 
by radiation of reference spectral regions, and a circuit for using the outputs 
corresponding to the different spectral regions to remove visibility of the different spectral 
regions from the sensor output, to permit high resolution to be attained. 

2. A camera as claimed in claim 1, including a waveform generator for generating 
pulses derived from the stored signals which are applied to sample and hold means which 
receive a signal derived from the sensor output. 

3. A camera as claimed in claim 1 or claim 2, in which the stored signals are square 
waves derived from the sensor output. 

4. A camera as claimed in any one of claims 1 to 3, in which the sensor is arranged 
to produce an output representative of lines of the image, and the filter means comprises 
stripes orientated obliquely to the lines of the image. 
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claimed in claim 4, in which the stripes are formed of two sets, one 
comprising stnpes passing one spectral region alternating with stripes passing all 

and the second set comprising stripes passing another 



5 A camera as 
set 

spectral regions of the sensor 
spectral region alternating with stripes passing all spectra! regions of the sensor. 

6 A camera as claimed in claim 5, in which the said one spectral region and the said 
o^er speora. region are sub-active primary colours, and the alternating stripes are clear. 



7, A camera as 
yellow and cyan. 



claimed in claim 6, to which the subtnKtive primary colours are 



8 A camera as claimed in any one of claims 5 to 7, in which the pitch of the snipes 
^ ». ahematog regions are equal, and each set of aripes is inclined a, a. angle to the 

line direction. 



o. Acamera as claimed in any one of claims 5 ,o 8, in which the sets of sunpes , 
inclined to different sides of ammonal vertical line of the image. 



10 A camera as claimed in an, one of claims 1 to 9, to which the reference spectra, 
regions are primary colours. 

, ■ A -, n riaim 5 in which the filter means lies at the image plane 
11. A camera as claimed in claim 5, in wrucn 

oftie sensor or at the focus of a relay lens for the sensor. 



12. A camera as claimed in any one of claims 1 to 1 1, in which the sensor includes an 
image intensifies 

13. A camera as claimed in claim 12, in which the sensor includes a solid state imager. 

14. A camera as claimed in any one of claims 1 to 1 1 , in which the sensor is an infra- 
red sensor. 

15. A camera substantially as herein described with reference to the accompanying 
drawings. 
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(57) Abstract 

The apparatus (10) is a color wide dynamic range apparatus which includes a filter (12) interposed immediately in front of reoccurring 
color elements so that each pixel represents a given color element for the scene. At least two exposure levels are taken of the scene and 
the pixel outputs are decoded to generate the video luminance and chrominance signals. The images of the at least two exposure levels are 
combined to form a final image. 
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Description of the Prior Art 

Various types of video enhancement apparatuses and 
techniques have been proposed. Prior implementations of 
color vide dynamic range cameras, such as those disclosed 
5 in the above-identified parent applications hereto, have 
used a plurality of CCD chips to generate the image data 
for subsequent processing. The use of multiple CCD chips, 
however, adds to the complexity and cost of the instrument. 
Moreover, current consumer video cameras, i.e., camcorders, 
10 almost universally use a single CCD chip. Therefore, a 
single CCD implementation is required to use dynamic range 
enhancement algorithms in a camcorder. Single CCD chip 
implementations are similarly preferred for endoscopic 
applications. 

15 

OBJECTS AND SUMMARY OF THE INVENTION 

It is therefore an object of this invention to provide 
a color wide dynamic range camera implemented with a single 
CCD chip. 

2 0 It is also an object of this invention to provide a 

color wide dynamic range camera which is adapted for use 
with a camcorder. 

It is a further object of this invention to provide a 
color wide dynamic range camera which is adapted for use 

2 5 with a conventional endoscope. 

These and other objects of the invention will be more 
apparent from the discussion below. 

SUMMARY OF THE INVENTION 

3 0 There is thus provided in accordance with the 

preferred embodiment of the present invention, video 
imaging apparatus including means for providing a plurality 
of video color images of a scene at different exposure 
levels using a single CCD chip, each color image being 
35 separated into several (e.g., four in the preferred 
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embodiment) different components prior to sensing by the 
CCD chip by way of a multiple color mosaic filter in front 
of the CCD chip. The pixel outputs are then decoded — 
subjected to specific mathematical operations by the 
5 processing electronics following the CCD output — to 
generate the video luminance and chrominance signals. 

The present invention integrates the digital 
processing of the mosaic color CCD data with ADAPTIVE 
SENSITIVITY™ dynamic range enhancement. This integration 
10 provides for a substantial savings in total system 
processing hardware chip count and cost. It also permits 
better control of the color and detail production of the 
camera's video output. The mosaic storage format also 
provides for a unique video image compression technique. 

15 

BRIEF DESCRIPTION OF THE DRAWINGS 
Further objects and advantages of the invention will 
become apparent from the following description and claims, 
and from the accompanying drawings, wherein: 

2 0 Figure 1 is a general block diagram of the present 

invention. 

Figure 2 is a representative illustration of the data 
image elements, with the size of the data image elements 
exaggerated . 

25 Figure 3 is a general block diagram of the long and 

short processing of the present invention. 

Figure 4 is a block diagram of the color path of the 
present invention . 

Figure 5 is a block diagram of the intensity path of 

3 0 the present invention. 

Figure 6 is a block diagram of the look-up table 
processing of the present invention. 

Figure 7 is a block diagram of the joint operations of 
the present invention. 
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Figure 8 is a block diagram of the differential color, 
intensity result block of the present invention. 

Figure 9 is a block diagram of the color suppression 
factor block of the present invention, 
5 Figure 10 is a block diagram of the color conversion 

block of the present invention. 

Figure 11 is a block diagram of the mosaic generation 
block of the present invention. 

10 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Referring now to the drawings in detail wherein like 
numerals indicate like elements throughout the several 
views, one sees that Figure 1 is a block diagram of the 
apparatus 10 of the present invention. 

15 Apparatus 10 includes a mosaic filter 12 which is 

bonded to the front of CCD 14 (preferably a single chip) , 
generally as part of the CCD integrated circuit 
manufacturing process. 

The alternating mosaic filter elements are cyan, 

20 magenta, yellow and green (wherein C 'cyan' = G 'green' + 
B 'blue'; M 'magenta' = R 'red' + B; and Ye 'yellow' = 
R+G) . When the CCD 14 charge output is read out, the 
photoelectric charges from vertically adjacent sensor 
elements of CCD 14 are combined in the analog shift 

25 register (not shown) . The on-chip addition gives rise to 
a, 0, 7 and 6 elements as described below and as described 
in the Sony CCD 1992 Data Book (Sony part number ICX03 8AK) , 
as well as earlier editions. 

As shown in Figure 2, the mosaic complementary 

3 0 additive color image comprises alternating first and second 
rows of image data elements 18 A — 18 D wherein first rows 
include alternating a and y data elements (18 A and 18 B , 
respectively) , and wherein the second rows include 
alternating j8 and S data elements (18 c and 18 D , 

3 5 respectively) . 

SUBSTITUTE SHEET (RULE 26) 
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The a image data elements 18 A are an equal mixture of 
cyan plus green (i.e., C + G = B 'blue' + 2G 'green'). The 
y image data elements 18 B are an equal mixture of magenta 
plus yellow (i.e., M + Ye = 2R 'red' + B + G) . Similarly, 
5 the $ image data elements 18 c are an equal mixture of cyan 
plus magenta (i.e., C + M = 2B + G + R) and the 6 image 
data elements 18 D are an equal mixture of green plus yellow 
(i.e. , G + Ye = 2G + R) . 

Those skilled in the art will recognize that: 

10 Y (i.e., intensity) = = 

from which the definition of intensity (Y) in the red, 
15 green, blue (RGB) system may be derived: 

Y=R+1.5G+B 

Of course, those skilled in the art will realize that 
other color combinations are equally applicable. Each 
mosaic element of filter 12 covers the sum of two adjacent 

2 0 pixel sensors of CCD 14 so that each pixel output , of CCD 14 

is representative of one of the above color combinations 
given for the various image data elements 18. Four 
different monochromatic images, each representative of one 
color combination chosen from the colors of image data 
25 elements 18 A - 18 D of a given scene, are therefore generated 
by CCD 14. 

As can be further seen from Figure 1, apparatus 10 
includes four major functions as summarized below: 

1. Long/short exposure processing: 

3 0 The first stages of the algorithm are performed on the 

two exposures (long/short) separately. The processing 
of each exposure is divided into two paths: 

a. Color path processing - evaluates color component 
for each pixel. 

35 

b. Intensity (Y) path processing - handles intensity 
information for each pixel. This includes point 
("DC") intensity and edge information. 

SUBSTITUTE SHEET (RULE 26) 



BNSDOCID: <WO 941 8801 A1 > 



PCT/US94/01358 

6 

Point processing: 
Each of che long/ short exposure length processing 
functions (typically implemented on separate chips) 
outputs its point intensity information, obtained from 
the Y path processing, to four look-up tables (LUTs) . 
These tables determine the point intensity result of 
the two exposures, the normalized color weighting or 
color selection function and the saturation color 
suppression factor. This information serves the joint 
operation processing stage. The four LUTs are 
programmable, thus enabling operation with different 
functions when necessary. In an alternative embodi- 
ment, these LUTs may be replaced by a programmable, 
piecewise linear (PWL) or other digital function 
generator . 

Joint operations processing: 
Joint operations processing joins results produced by 
the long and short processing blocks, and results 
obtained from the functions implemented in the table 
processing LUTs, and evaluates the final output of the 
algorithm. The processing is divided into: 

a. Color components and Y result calculation — 
evaluates the final result of the color 
components and the intensity of each pixel. 

b. Color suppression factor calculation — evaluates 
the color suppression factor for each pixel, 
based on both edges and saturation information. 

c. Color conversion processing — converts mosaic 
differential color space to RGB color space and 
produces RGB and Y/Cr/Cb outputs for each pixel. 

Generate mosaic processing. 

Generate Mosaic processing converts RGB color space 

back to mosaic color space for each pixel. The mosaic 

information generated enables economical hardware 

storage of processed images. This information can be 

retrieved and replayed through the algorithm — in 
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Replay Mode — to produce RGB or Y/Cr/Cb output of the 
stored result. 

Referring now to Figure 1, similar to U.S. Patent No. 
5,144,442 and parent U.S. patent application Serial No. 
5 07/795,350 (the disclosures of which, again, along with 
U.S. patent application Serial No. 07/805,512 and U.S. 
Patent No. 4,858,014 are incorporated herein by reference), 
apparatus 10 includes long/short processing as implemented 
by mosaic long exposure field block 20 and mosaic short 

10 exposure field block 22 which obtain, respectively, a long 
and a short exposure from CCD 14 in order to allow subse- 
quent processing by long exposure processing block 24 and 
short exposure processing block 26. The terms "long" and 
"short" exposures are used here generally to denote two 

15 image inputs to apparatus 10. In general, "long" is used 
to mean an input with a higher exposure level, and "short", 
a lower exposure level. The higher exposure may be 
generated in several ways, including longer integration 
time, typically obtained by controlling the "electronic 

2 0 shutter" of the CCD chip; higher gain in the analog 

amplifiers preceding digitization; or a larger mechanical 
iris opening or other external gating means. 

These two image inputs are usually generated by a 
single CCD chip, but may also be generated simultaneously 
25 by two separate, bores ighted CCD chips, as disclosed in the 
aforementioned earlier applications. For the more common 
case in which the two inputs are generated by a single CCD 
chip, they may be generated either sequentially (as in the 
case of the first method above — integration time control) 

3 0 or concurrently (by using two input channels with different 

gain levels) . When a sequential method is used, field 
memories are required at the input to apparatus 10 (in 
blocks 20 and 22) to synchronize the data coming from the 
two sequential fields or frames. These memories are not 
35 needed in concurrent modes, except for purposes of 
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"freezing" the image for electronic, digital storage • 
Switching logic incorporated in blocks 2 0 and 22 controls 
the data flow into and out of these field memories, 
depending on which mode (sequential or concurrent) is used* 
5 Of course, this implementation could be expanded to more 
than two exposure levels. Blocks 24 and 2 6 may typically 
be provided on separate processing chips or incorporated 
together in a single chip. The processing for each 
exposure is divided into two paths: 
10 l. Color path processing — handles color information 

for each pixel (see color path block 28 in Figure 
3 and, in more detail, in Figure 4) ; and 
2. Intensity (Y) path processing — handles intensity 
information for each pixel (see Y path block 3 0 
15 in Figure 3 and, in more detail, in Figure 5) . 

Additionally, as shown in more detail in Figure 3, 
long/short exposure processing blocks 24, 26 include mosaic 
white balance block 32. 

Mosaic white balance block 3 2 receives the following 
20 field of information from long/short exposure field blocks 
20, 22: 

a 7 a 7 a 7 .... 

8 6 0 6 0 6 

a 7 a 7 a 7 .... 

25 0 6 0 6 0 6 

a 7 a 7 a 7 ... . 

0 6 0 6 0 6 

... 
... 

30 ... 

That is, the information from CCD 14 with the mosaic 
data order intact is received. 

After processing, the mosaic white balance block 3 2 
outputs color-corrected data values: 

SUBSTITUTE SHEET (RULE 26) 



BNSDOCI0:<WO 9416801A1> 



WO 94/18801 PCT/US94/01358 



a wb Twb a »b 

Pwb &wb £wb 

a wb 7wb <*wb 

£wb *wb £wb 

^wb Twb a wb 

£wb ^wb £wb 



9 

Twb a wb Twb * 

^wb ^wb *wb * 

Twb a wb Twb • 

^wb £wb ^wb • 

Twb Q wb Twb • 

^wb £wb *wb • 



10 Mosaic white balance block 32 contains mosaic color 

balance functions. These functions may typically be 
implemented as eight mosaic white balance LUTs (look-up 
tables) . That is, for each exposure there is a set of four 
LUTs, one for each mosaic data type: oc, y, and 6. 

15 Independent calculation of white balance correction factors 
is performed for each exposure. This enables white 
balancing scenes where the observable parts of the two 
exposures are at different color temperatures. The LUTs 
may contain multiplicative correction factors which are 

2 0 evaluated as follows: 

From the definitions of a, /?, y, S and Y it follows, 
that for a white image (where by definition R=G— B ) , the 
following relations should hold: 



25 


<*wh 








-f.T 


30 


Twh 




35 
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35 



40 



45 



50 
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Based on these relations correction factors can be 
calculated by enforcing these relations on the average of 
a white image: 

a a 
8 
7 



10 8 , 



% = 



0 

15 8 

7 



* Y 



7 

20 6 



* Y 



25 where Y denotes a selective average over Y in the given 
white image and or, fi f y and S are the respective average 
values of ql, /J, y and 6. Saturated or cutoff pixels are 
excluded from this average. Since by definition, 



a 


+ 


7 8 + <S 




2 


2 


■ the 


correction : 


3 


* 




7 
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* 
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The LUT values are calculated by simple multiplication of 
each mosaic data type by its respective correction factor: 
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<*wb 
7wb 



a * C a 
0 * c, 

$ * C 5 



In an alternative embodiment, these LUTs are replaced 
by digital multipliers. Furthermore, the LUTs may also be 
loaded with correction functions other than simple linear 
multiplicative factors • Alternatively, the mosaic balance 

10 correction factors can be computed based on four average 
signals, namely, a, £, y and S, instead of merely two of 
them as above. This alternative yields improved uniformity 
of color balance under difficult conditions. Alterna- 
tively, the white balance function may be done on the RGB 

15 color components in the color conversion block 78 
(described below) . 

Referring now to Figure 4, color path block 28 is 
shown in more detail. As previously stated, the input to 
color path block 28 is the image data a wb , 0 wb , 7 wb , <S wb after 

20 processing by mosaic white balance block 32. 

The initial processing of color path block 28 is 
performed by color difference evaluation block 34 which 
receives data a wb , /3 wb , 7 wb , 6 wh from mosaic white balance 
block 32 and calculates color difference components dr, db 

25 for each pixel in the array: 



dr dr dr . . . . = (7-a) (7-a) (7-a) .... 

db db db . . . . = (6-0) (6-0) (6-0) .... 

dr dr dr . . . . = (7-a) (7-a) (7-a) .... 

db db db . . . . = (*-£) (S-0) (<$-£) .... 

3 0 wherein: 

dr s 7 - a , the differences between successive 
readings in even lines. 
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db = S - )8 , the differences between successive 
readings in odd lines. 

The correct evaluation of dr and db requires 
horizontal interpolation as described in the following 
5 equations: 

For each pixel (i,j) in even lines i: 

<*wb(jodd-l) + <*wb(jodd+l) 

( j^) =7wb( jodd) " 2 

10 7wb(3cv«-D + Ywb(jcvc*+1) , 

(Devm)- 2 *wb(Dw«) 

where j is the pixel index along the line (here and 
henceforth) . 
15 For each pixel (i,j) in odd lines i: 

£wb(jodd-i) + £ wb (jodd+D 



db (jodd) = ^wb( Dodd) " 2 

20 SwbCjcvca-l) + ^wb(jcvcn+l) 0 % 

= ^ tfwbOevcJ 

Color difference components dr, db are thereafter 
2 5 received by low-pass color component block 3 6 which 
calculates a low-pass color component dr^ or db^ for each 
pixel: 

drjpdr^dr^ . . . . = (y-cx)^ (y-a) ^ . . . . 
db^db^db^, • . . . = (5-/8)^ . . . . 

30 dr^dr^dr^ . . . . = (7-*)* (7-<*)ip .... 

db^db^db^ . . . . = (*-£)ip (S-jS)ip . . . . 

Block 3 6 performs horizontal low-pass filtering on dr^ 
and db^ calculated in block 34. This reduces color 
artifacts caused by interpolation. In a preferred 
35 embodiment, the low-pass filter width is five pixels and 
its coefficients are Vt, hi h, h, Vi. The equations follow: 
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For pixels (i,j) in even lines i: 

dr(j-2) + 2*dr(j-l) + 2*dr(j) + 2*dr(j+l) + dr(j+2) 
dr lp (j)= 



For pixels (i,j) in odd lines i: 

db(j-2) + 2*db(j-l) + 2*db(j) + 2*db(j+l) + db(j+2) 



db lp (j) = 



Delay buffer 38 receives the output from low-pass 
color component block 3 6 and directs db^i^ -1) , dr lp (i 0(Id - 
1) , db^i,^, +1) and dr^i^H-l) to vertical interpolation 
block 40 and drCi^,,,,) and dbCi^) to multiplexer 42. 
15 Vertical interpolation block 40 receives the low-pass 

color components as described above and generates 
interpolated low-pass color components dr',,, in the odd 
numbered lines and db'^ in the even numbered lines: 

db',,, db',, db'„, db',p 

20 dr',, drV drV dr'^, 

db^ db'.p db'„, db',, 

dr',, dr'„, dr'„, dr'* 

The equations follow: 

For even lines i: 

25 db,p(i-l,j) + db,p(i+l,j) 

db'*(i f j)- : = 



For odd lines i: 

dr,.(i-l,j) + dr,_(i+l,j) 



30 dr' (i,j) = 
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The interpolated low-pass color components dr'^, db'^ 
are multiplexed with the original low-pass components dr^, 
dbjp to give the color path output values dr and dp for each 
pixel. This function is performed by multiplexer 42 , which 
5 separates the output received from delay buffer block 38 
and vertical interpolator block 4 0 into a first path 
including db,p' (icv«) and db^i^) and a second path including 
dr,p (i CTal ) and dr'^i^) . 

Referring now to Figure 5, which discloses in more 

10 detail the intensity (Y) processing block 3 0 shown in 
Figure 3, one sees that the input to intensity (Y) 
processing block 3 0 from mosaic white balance block 3 2 
(Figure 3) is received by intensity evaluation block 44 
which outputs computed intensity Y for each pixel. 

15 Since only one of the four data types (a, 0, y, 6) is 

present at any given pixel, the intensity evaluation block 
44 calculation is performed as follows (based on the prior 
definition of Y) : 

For pixels (i,j) in even lines i: 

20 



25 



30 



Y(jodd) 

For pixels (i,j) in odd lines i 

Y(jevcn) 



70) 


+ 


a(j-l) 




2 






+ 






2 




lines 


i : 




Hi) 


+ i8(j-l) 




2 




00) 


+ 





The output from intensity evaluation block 44 is 
3 5 received by delay buffer 46, generate output intensity 
block 48 and limit block 50. 

Delay buffer 46 is a delay line of two horizontal 
lines, required for the 3x3 and 1x3 matrix transformations 
in Y path block 30. Together with the color path delay 
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buffer 38 and with Y path delay buffer 54, it may be 
implemented in a preferred embodiment in mosaic data. space, 
operating on the input a, /?, 7, S data before the intensity 
(Y) evaluation block 44 and color difference evaluation 
block 34. It is shown here schematically for clarity. 

Vertical low-pass filter 52 receives intensity (Y) 
signals from the intensity evaluation block 44 as delayed 
by delay buffer 46. Block 52 generates the vertical low- 
pass intensity Y^ defined as: 



The unfiltered intensity (Y) input will sometimes 
exhibit horizontal stripes, one pixel high in each field, 
in areas of transition to saturation. These stripes stem 
from the different color spectra of the a, £, 7, and S 
pixels, as a result of which the at+7 value of Y(i(even),j) 
may, for instance, reach saturation at a lower level of 
optical intensity than the fi+6 value of the vertically 
adjacent Y (i+1 (odd) , j ) . Y vlp averages these values to obtain 
a function that is smooth over the transition area. 

Generate output intensity block 48 receives intensity 
(Y) information from intensity evaluation block 44 and 
vertical low-pass intensity (Y vlp ) information from vertical 
low-pass filter 52. The output of block 48 is output 
intensity (Y^) to point processing LUT block 62 (see Figure 
1) . 

Block 48 replaces the original luminance Y, computed 
by the intensity evaluation block 44, with Y vlp when Y vlp 
approaches saturation, in order to prevent the appearance 
of horizontal stripes as explained above. Block 48 
implements the function: 



Yv*(i, j) 



= Y(i-l,j) + 2Y(i,j) + Y(i+l,j) 



4 
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— | Y if Y vlp < Y^jj^j^y 

I Y vlp 1 f Y vip > Y^jijoy 

The value of Y^ue,^ is typically equal to approximately 
5 220 on an 8-bit scale of 0-255. As values of Y approach 
saturation, image detail is lost in any event, so that 
substituting Y vlp in the high range does not adversely affect 
the perceived resolution* Y V |p is used as the selecting 
input in order to ensure a smooth transition* 

10 Limit block 50 receives intensity (Y) signals from 

intensity evaluation block 44 and generates limited 
luminance Y^. Limit block 50 cuts off the upper range of 
intensity (Y) values that are to be input to edge detection 
block 56, in order to prevent detection of false edges or 

15 horizontal stripes that can arise in areas of transition to 
saturation. Limit block 50 implements the function: 

Y^ = min {Y, Y^} 

the value of Y^ is typically equal to approximately 22 0. 

The output of limit block 50 (i.e., Y^) is delayed by 
2 0 delay buffer 54 and received by edge detection block 56 
which outputs edge information for each pixel. 

Edge detector block 56 convolves the Y^ value and its 
8 immediate neighbors, with a high-pass or edge detecting 
kernel . 

2 5 In one embodiment, the 3x3 Laplacian operator may be 

used: 

-1 -1 -l 1 
-1 8 -1 
-1 -1 -1^ 

Alternatively, to accommodate the geometric 
characteristics of the CCD raster and to give greater 
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emphasis to the vertical edges, the following kernel may be 
used: 

1 f"* - 1 
5 -5-* -2 8-2 

In an alternative embodiment, the edge detector block 
56 could be implemented as separate horizontal and vertical 
10 convolution operations (such asalx3or3xl matrix) , 
with additional logic to avoid overemphasis of diagonal 
edges. This alternative embodiment is less hardware 
intensive and gives improved picture quality in some 
circumstances . 

15 Edge suppress block 58 receives the vertical low-pass 

intensity (Y vlp ) signals from vertical low-pass filter 52* and 
outputs edge suppression function f^ to edge multiplier 60* 
The edge suppression function varies between 0 and 1 
in the long exposure processing block 24 only. In the 

2 0 short exposure processing block 26, the function is set to 
1, i.e., no edge suppression at this point. The function 
is typically implemented in block 24 in a piecewise linear 
fashion as follows: 



§ Y vlp - LOWSAT 

f«tec 5=5 1 1 - - xf LOWSAT < Y vto < DEEPSAT 

^ 1 DEEPSAT— LOWSAT * 



if Y vJp < LOWSAT 

^ LOWSAT < Y vlp < 
if DEEPSAT < Y 



vlp 

3 0 Typically LOWSAT is set to approximately 190 and 

DEEPSAT to approximately 220. 

Edge multiplier 60 receives input from blocks 56, 58 
and generates suppressed edge ed^ to intensity (Y) result 
calculation. 

3 5 Edge multiplier 60 multiplies the edge output of the 

edge detector block 56 by the edge suppression function f^ 
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from block 58 to generate an output value ed^ to joint 
operations block 64 (see Figure 1) . The purpose of this 
multiplication is to suppress distorted large edges that 
may appear in the long exposure at intensity (Y) values 
5 near saturation, at the same time as they appear in the 
short exposure at lower values of intensity (Y) • The 
double appearance of such edges was found empirically to 
cause the resulting displayed edges to be overemphasized 
and sometimes smeared on account of blooming in the long 

10 exposure. The long exposure edge is suppressed so that 
only the short exposure edge will pass through to the 
output image. The edge suppress function may also be used 
to reduce the amplitude of edges from the long exposure 
which may be otherwise exaggerated due to the higher gain 

15 of the long exposure relative to the short exposure. 

Additionally, as shown in phantom in Figure 5, an 
optional multiplier or LUT (block 57) may be added to 
multiply the output of block 56 times the ratio of exposure 
times (duration of long exposure/duration of short 

20 exposure) or the corresponding gain ratio, or some function 
of the exposure and/or gain ratio. This reflects the ratio 
of scales of these two values. 

In the above manner, Y path block 3 0 outputs processed 
luminance Y^, edge, and edge^ to point processing block 62 

2 5 and joint operations block 64. 

Referring now to Figure 6, one sees that point 
processing block 62 includes four point processing 
functions, all of which receive output intensity (Y^J 
values from the long and short exposure processing blocks 

30 24, 26 (see Figure 1). These functions may typically be 
implemented as LUTs in RAM or ROM memory. Point processing 
block 62 generates arbitrary function values for input to 
the joint operations block 64 (Figure 1) . 
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The four tables of block 62 are: 

1. The intensity (DC result) block 66 which 

generates a LUT value of intensity (Y^) for the joint 
operations block 64. 
5 Block 66 controls the amount of point ("DC") 

luminance that is summed with the edge information in 
generating the output luminance, Y^^. In its most 
general formulation, 

io y.* = f(vr 

where f is an arbitrary function. It has been found 
that a quasilogarithmic or fractional power dependence 
of Yh* on the inputs gives the best output image 
appearance, and the general function above can 
15 generally be reduced to a more compact LUT or 

piecewise linear implementation. 

One simple possible computation of Y^ is as 

follows: 

a ) Y thoit is multiplied by the exposure ratio, so that 
20 it is on the same scale as Y^. That is, if a 

certain pixel x is acquired within the active 
(linear) sensitivity region of both the short and 
long exposures, then Y loog (x) = R*Y lhort (x) , where R 
is the exposure ratio, R=long exposure time/ short 
25 exposure time (or any other ratio representing 

the two sensitivities) . 

b) Subsequently, Y^ and R*Y Aort are linearly 
combined, so that the sum of their relative 
weights is always 1. That is, Y wdr =a*Y kJCg + 

3 0 b*R*Y tbort , l>a>0, l>b>0, and a+b=l (the wdr index 

stands for 'wide dynamic range'). The common 
practice is to set a=l in the region where the 
short exposure is cut-off (too dark) , b=l in the 
region where the long exposure is saturated (too 
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bright) , and a>0, b>0 in the region where both 
exposures carry meaningful information. However, 
this does not cover all cases, e.g. when neither 
exposure carries any information (long is 
5 saturated and short is cut-off, or both 

saturated, or both cut-off) . 
c) Finally, the dynamic range of Y wdr is reduced 
(yielding Y^) by either a logarithmic function, 
or by multiplying it by a small fraction, or by 
10 using any empirically found mapping which 

resembles the log function or a similar contrac- 
tion, e.g., the square root. 
Other possible values for Y^ comprise empirical 
modifications of the function described above. 
15 2. & 3. Color weight normalize blocks 68, 70 for 

long and short exposures, respectively, which 
generate normalizing color weights Wj/Y, and w g /Y t . 
Blocks 68 and 70 control the proportions of 
mixing the color values, dr and db, from the long 

2 0 and short exposures, respectively, that will be 

used to generate the output color values, dr,^ 
and db,^. Generally, w, and w, are chosen so as 
to give predominant weight at each pixel to the 
color values taken from the exposure in which the 
25 intensity (Y) luminance values are in the linear 

portion of the range, and to give a smooth 
transition over luminance gradient regions of the 
image. For the most part, w { and w, are deter- 
mined on the basis of Y^flong) alone, except for 

3 0 cases where the long exposure is near saturation 

while the short is near cutoff, so that neither 
gives a linear reading. 

The weighting values are complementary, 
i.e., w, = 1 - w a , and w,, w, > 0. The outputs of 
35 blocks 68 and 70 are normalized by division by 
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the corresponding values of for the long and 
short exposures* Preferably, a floating point 
representation for the output values of blocks 
68, 70 is used so as to maintain sufficient 
accuracy to prevent noticeable quantization in 
the output image. 

Alternatively, instead of weighted addition 
of the normalized color from the two exposures, 
simple selection of the normalized color from one 
exposure or the other, with an ordered dither 
(alternation) of the color selection in areas of 
transition may be used. In this case, when w, = 
1, the color is selected from the long exposure; 
when w t = 1, it is selected from the long 
exposure, and when neither w factor is 1, color 
values are taken alternately from long and short, 
according to a pseudo-random probability 
distribution in which the long and short color 
value probabilities are equal to the w { and w, 
values. A normalized color value that is an 
average of the long and short values may also be 
mixed into the dither, in order to give a 
smoother color transition. 

Saturation color suppression factor block 72 
generates the color suppression factor Wht that 
reduces chroma saturation (adds white to the 
image) in areas of luminance saturation of the 
input image. An additional edge color 

suppression factor, Z^, is computed in the joint 
operations block (as will be described herein- 
after) . The minimum of Wht and Z^, both of which 
vary from 1 to 0, multiplies the chroma compo- 
nents at the output stage of color conversion. 
Thus, as Wht approaches zero, so does the color 
saturation of the output image. 
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The purpose of the saturation color 
suppression function is to reduce the appearance 
of color artifacts that arise due to CCD 
saturation. The linear relationships between the 
5 a, £, 7, and S CCD outputs and the true RGB 

colors break down as the CCD 14 approaches 
saturation. As non-linear deviations cannot be 
readily corrected, suspected distorted colors are 
"whitewashed". Similar techniques are used in 
10 the analog domain in conventional CCD cameras. 

As shown in Figure 6: 

Wht = w, + w,*z, 

w x and w g are identical to the above color 
weighting values. The variable Z, is a function 
of Y^ ( short ) , varying between 0 and 1, as shown 
schematically in the lower right corner of Figure 
6. It tends to zero in areas where the short 
exposure luminance approaches either saturation 
or cutoff. This function will give Wht - 0 at 
the saturation end (where generally w § = 1 while 
Wj = 0) . At the cutoff end, normally w, » 1 as 
long as there is adequate overlap between the 
long and short exposures, so that in this range 
the function will usually give Wht « i. 

In normal mode, in which only one input 
channel is operative (see explanation below) , Wht 
= 1 from Y=0 up to the low saturation threshold 
(typically 190) . From this threshold up to the 
deep saturation limit of Y (typically 2 20) , Wht 
drops linearly to its saturation value of 0. In 
replay mode (see below) , there is no saturation 
color suppression. 

Figure 7 discloses the joint operations block 64 (also 
Figure 1) . 
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Joint operations block 64 combines the chrominance and 
luminance data from the long and short exposure processing 
blocks 24, 26, together with data from point processing 
block 62, to generate a combined Y/dr/db result. Block 64 
5 then converts this result to output in standard RGB or 
Y/Cr/Cb (luminance, chrominance (red) and chrominance 
(blue)) color space. A color suppression factor Z is 
computed and applied to the chrominance outputs in order to 
reduce color artifacts (by reducing chroma saturation) 
10 around edges and areas of luminance signal saturation. 

Joint operations block 64 includes: 

1. The dr, db, Y block 74 (recalling that dr 
and db are the differences between 
successive readings in even and odd lines, 

15 respectively) which receives dr, db values 

from the color path outputs of long and 
short exposure processing blocks 24, 26 
respectively; ed^ from the intensity (Y) 
path output of long exposure processing 

2 0 block 24 and edge data from the intensity 

(Y) path output of short exposure processing 
block 26; and Y LUT , w x /Y { and w./Y, from table 
processing (LUT) block 62. Block 74 
generates combined intensity Y/dr/db results 
25 to color conversion block 78 (to be 

discussed) . Block 74 will be discussed in 
greater detail hereinafter. 

2. The color suppression factor block 76 which 
receives ed^ and ed,^ from edge detector 

3 0 block 56 and saturation color suppression 

factor (Wht) from point processing block 62 
and generates chroma suppression factor 2 
for color conversion block 78. Block 76 
will be discussed in greater detail 
35 hereinafter. 
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3. The color conversion block 78 which receives 
^ttMuhi .<**W from block 74 and Z, the 
color suppression factor from block 76 and 
generates R^, G^, and and Cr and Cb. 

5 Block 78 will be discussed in greater detail 

hereinafter. 

The dr, db, Y block 74 is shown in further detail in 
Figure 8. 

Block 74 includes an intensity (Y) calculation which 
10 is performed by adders 79, 80 and edge limiting block 81. 
Adder 79 receives ed^ (long) data from long exposure 
processing block 24, and ed,^ from short exposure 
processing block 26. These two inputs are added to give 
ed 9 e «*iit/ which is then input to the edge limiting block 81. 
15 Edge limiting is implemented as a piecewise linear function 
with 6 inflection points (A, . . . A$) and 4 slopes (Si...S 4 ), as 
shown in the upper right inset of Figure 8 . Generally the 
inflection points and slopes are chosen so as to enhance 
the smaller edges (i.e., S 2 and S 3 > 1) , while large edges 

2 0 (edge > A s or < A 2 ) are suppressed. Since these large edges 

come through strongly in the Y LUT contribution anyway, the 
output image has a more pleasing appearance if they are not 
additionally enhanced. A 3 and A4 may be set to 0, but it is 
sometimes desirable to set them to small non-zero values in 
25 order to suppress false edges due to noise. The best 
results appear to be obtained with \a x \ and IaJ values of 
50 to 60. The best values of the slopes |S;| are typically 
in the range 0.5 to 2, but the hardware allows a greater 
range . 

3 0 The edge^,^ output is then summed by adder 80 with the 

Y LUT output of block 62 to obtain the output luminance value 
Y re«uif Additionally, as shown in phantom in Figure 10, adder 
80 may be removed from its location in Figure 8 and placed 
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so thac the output of block 81 is not added to Y,^ until 
just before being added into block 113 A<: , that is, as late 
as possible. 

5 Block 74 further includes a dr, db calculation which 

is performed by the remaining sections of block 74. The 
dr, db calculation receives low-pass color components dr, 
db from the color paths of long and short exposure 
processing blocks 24, 26; w,/Y, and w 8 /Y g from block 62; and 

10 Y re«it as calculated by adder 80. The dr, db calculation 
outputs dr^ and db^. 

The long and short values of dr and db are multiplied 
by the respective normalized color weights, w^Y, and w 8 /Y 8 by 
multipliers 82, 84, 86, 88. These normalized, weighted 

15 color values from the two exposures are summed together by 
adders 90, 92 and then multiplied by Y,^^ by multipliers 94, 
96 to give the scaled values: 

20 drrcauh = YnKlk * l dri + dr ' 

db^ = Y^ * (db, 2- + db, 

I Y, Y, i 

Alternatively, dr,^ and db,^ may be generated by 
25 selection between the long and short normalized dr and db 
inputs (and possibly their long/short average values) . 

The color suppression factor block 76 of Figure 7 is 
shown in more detail in Figure 9. 

Maximum value block 100 selects the higher of the two 
3 0 absolute values of ed^ and ed,^ as calculated by absolute 
value blocks 98, 99. The result of the calculation of 
block 100, ed^, is input to edge chroma suppression factor 
block 102 to calculate Z^. The calculation of Z^ is 
implemented as a piecewise linear function, shown in the 
3 5 upper right corner of Figure 9. As can be seen in Figure 
9, Z^ receives a value between Th and 1, given by: 
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Th + 



Th 



1 



E2 - ed^ 



E2 - El 



(1 



Th) if El<ed nux <E2 



if E2<ed muc 



if ed m4X <Ei 



Typically, El=10 and E2=27 have been found to give 
good results. The minimum value of Z^,, Th, is ordinarily 
set to zero, to give complete chroma suppression at very 
strong edges. Th ^ 0 is used only in replay of images 
stored in mosaic format (see generate mosaic block 120 
described hereinafter) , in which case serves to suppress 
color anomalies resulting from the reinterpolation of the 
pixel values. 

Thereafter, as shown in Figure 9, minimum value block 
104 selects the minimum of the two color suppression 
factors, Z^ and Wht, thereby determining the edge criterion 
or saturation criterion that should be used to provide the 
required degree of chroma suppression at the given pixel. 

Referring now to Figure 10, which discloses in detail 
color conversion block 78 of Figure 7, one sees that color 
conversion block 78 receives Y,^, dr,^, and db,^^ from block 
74 and Z from block 7 6 and generates outputs in both the 
RGB and Y/Cr/Cb formulations. 

In other words, block 78 takes the interim dynamic 
range enhancement results Y/dr/db, and converts them into 
conventional color components for system output. 

Block 78 includes horizontal low-pass filter 106 which 
receives Y,^ and calculates Y wull (Ip) for the color matrix 
block 108. 

Horizontal low-pass filter 106 is identical to the 
low-pass color component block 3 6 in the color path block 
28 (see Figures 3 and 4) . Since the dr and db inputs to 
the color matrix 108 have already been low-pass filtered by 
this low-pass filter operator, it is necessary to filter 
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the intensity (Y) value as; well in order to prevent color 
artifacts. 

Color matrix block 108 receives Y,^ (lp) from 
horizontal low-pass filter 106 and dr,^ and db,^ from block 
74 and generates low-pass RGB color component outputs. 

If one recalls the derivation of Y, dr and db from the 
original a, 0, y and 6 values of the mosaic CCD input: 



10 



15 



Y = 



a + 7 _ + S 
2 2 



dr s 7 - a 
db s s - 0 

together with the RGB equivalencies of a, )8, y and 6, one 
obtains the following relationships between RGB and the 
Y/dr/db values: 



20 



R 
G 
B 



= 3.5 * 



0.2 0.4 0.1 
0.4 -0.2 0.2 
0.2 -0.1 -0.4 





' Y " 




dr 


J 


db 



25 



The factor of 3.5 is required for normalization of .the 
relation Y=R+i.5G+B. Due to hardware implementation 
considerations , the color conversion matrix is calculated 
as follows: 





R " 




1.0 


2.0 


0.5 " 




' Y ' 




G 


= 0.7 * 


2.0 


-1.0 


1.0 




dr 


30 


,B 




.1.0 


-0.5 


-2.0 




db 



35 



In this way the matrix multiplication is performed by 
a series of shift/add operations. The multiplicative 
factor 0.7 is combined (by multiplication) with externally 
programmed RGB white balance correction factors as is 
described hereinafter. 
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RGB white balance multipliers 109 A/ 109 B , 109 c receive 
low-pass RGB signals from color matrix block 108 and 
generate normalized low-pass RGB signals. 

Multipliers 109 A , 109 B , 109 c multiply each of the RGB 
5 low-pass values by a pre-computed white balance correction 
factor, adjusted by the normalization factor 0.7 required 
by the color matrix calculation. Although conventional RGB 
white balancing uses only two multiplicative factors, 
correcting R and B while G is held constant, this "short 

10 cut" does not preserve constant Y achromatic luminance. 

This loss of normalization may lead to the appearance of 
artifacts and incorrect luminance in the output. It is 
necessary, therefore, to use three multiplicative factors, 
normalized to preserve constant luminance Y. 

15 The calculation of the correction factors is performed 

off-line by capturing a white image and selectively 
computing average values R, G, B, and Y, excluding pixels 
near saturation or cutoff. From the definition Y = R + 
1.5G + B, it follows that for a corrected pixel in the 

2 0 white image, it should be found that: 



R - G - B - ITS Y 

From this relationship one derives the correction factors 
25 to be used in multipliers 109 A , 109 B , 109 c : 

l y 

R factor x 1 



3 0 G factor - 



B factor = 



3.5 R 
1 Y 



3.5 G 
1 Y 



3.5 B 



3 5 Output signal enhancement block 110 (which includes 

chroma suppression and RGB output functions) receives 
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corrected low-pass RGB color component signals from color 
matrix block 108 via multipliers 109 A/ 109 B/ 109 C ; Y,^ from 
block 74; Yj^ufc (lp) from block 106; and chroma suppression 
factor Z from block 76. 
5 As noted above, the RGB values output from color 

matrix block 108 are low-pass values. High-frequency image 
information is "re- injected" into RGB according to the 
following equation (given here only for the R component, 
since the treatment of G and B is identical) : 

10 Rhp « Rtp + K*Y rciuft - K*Y^S 

K is an arbitrary constant between 0 and 1, chosen 
according to the degree of high-frequency enhancement 
required. Values in the range 0.4 < K < 0.8 are typically 
15 used. 

The addition and subtraction of Y riMJil values to the RGB 
components can alter the original values of R/G and B/G, 
with the result that the correct hue of the image is not 
preserved. Therefore, in an alternative embodiment, R, G 
2 0 and B are multiplied by a high-pass enhancement function. 

Since the RGB color component values contain both 
luminance and chrominance information, the chroma 
suppression factor, Z, is best applied to chrominance-only 
components, by adders 113 A , 113 B , 113 c : 



25 


Cr = 


R 


YjTggjjft 




Cg = 


G 


~ ^rcsuh 




Cb = 


B 


YfTJUJl 



Combining these equations with the previous ones for "high 
frequency re-injection", one obtains the following formula, 
3 0 which is implemented as shown in Figure 10 (including 
arithmetic element blocks 114, 115, 116) to obtain Cr from 
Rip • 
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Cr = + .K*(*«* - - Y result 

and likewise for Cg and Cb. These Cr/Cg/Cb values are 
multiplied by Z by multipliers 112 A , 112 B/ 112 c . At this 
5 point, Y/Cr/Cb output is available directly (using Y = 
Ynaun) , though it is preferable to add a bias of +128 to the 
signed digital outputs Cr and Cb in order to convert them 
to positive values for D/A conversion. In the alternative, 
Y rc«it can be added back into the chroma-suppressed Cr/Cg/Cb 

10 values (by adders 113 A -113 C ) to obtain the final Rout/Gout/Bo*. 

Referring now to Figure 11, which discloses generate 
mosaic block 120 of Figure 1 in more detail, one sees that 
the input of generate mosaic block 120 is 'R out /G out /B aiJt from 
color conversion block 78 of joint operations block 64. 

15 The output of block 120 is the equivalent a, 0, y, 6 values 
in the format: 

^cq Tcq Tcq • * • • • 

£cq 0cq 

®cq Tcq ^cq Teq * • • • 

In order to reduce memory requirements for image 
storage and to allow stored images to be replayed through 
the apparatus 10 for display, the final RGB values from the 
processed image are used to generate equivalent, simulated 
25 mosaic values of a, £, y, and <S. In this way, only eight 
bits per pixel of information must be stored, rather than 
the 24 bits of full output information. These mosaic 
values can later be replayed to regenerate the stored 
image . 

3 0 The simulated mosaic values are generated by the 

following matrix in matrix block 122, based on the color 
equivalencies given hereinabove. 
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The factor of hi that multiplies the matrix is used for 
reasons of hardware convenience — in order to ensure that 
a, /3, 7, and S do not overflow the range 0-255 of 8 bits. 
To maintain the normalization relations given hereinabove, 
the factor should actually be 1/3.5. Therefore, in replay 
mode, mosaic white balance block 32 is used to multiply the 
a, /S, 7, and S values back by 4/3.5 (=8/7) before 
reprocessing. Finally, multiplexer 124 selects which one 
of the four mosaic values to output for each pixel 
according to the table: 



Pixel ij 





j even 


j odd 


i even 


a 


7 


i odd 




S 



Apparatus 10 has three modes of operation: normal, 
adaptive sensitivity (AS) , and replay. 

1. Normal mode emulates the performance of a mosaic color 
CCD camera without adaptive sensitivity. In this mode only 
the long exposure portion of the pipeline operates. The 
processing functions are limited to decoding the mosaic 
input into conventional color components: Y/Cr/Cb or RGB, 
while additionally performing filtering operations for 
anti-aliasing, detail (edge) enhancement and chroma 
suppression where required. 
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2. Adaptive sensitivity mode uses all the resources of 
the processing pipeline to generate wide dynamic range 
images as described hereinabove. 

3 • Replay mode is required for displaying images that 
5 have been stored in RAM or disk. Apparatus 10 stores these 
images in a regenerated mosaic format in order to save on 
storage memory requirements. Replay mode is similar to 
normal mode, except that most of the enhancement operations 
are not performed: since the stored data have already been 

10 filtered once, it is for the most part not desirable to 
filter them again. 

The preceding specific embodiments are illustrative of 
the practice of the invention. It is to be understood, 
however, that other expedients known to those skilled in 

15 the art or disclosed herein, may be employed without 
departing from the spirit of the invention or the scope of 
the appended claims. 
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CLAIM S 

What is Claimed is; 

1. A color wide dynamic range video imaging apparatus 
comprising: 

5 sensor means for providing a plurality of color video 

images of a scene at different exposure levels; 

means for dividing each color video image into 
components ; and 

means for processing said components of each of said 
10 plurality of video images to produce a combined color video 
image including image information from said components of 
each of said plurality of color video images by applying 
neighborhood transforms to at least one of said components 
of each of said plurality of video images, 
15 wherein said means for processing includes means for 

calculating point intensity data for said each of said 
plurality of said video images. 

2. The color wide dynamic range video imaging 

2 0 apparatus of Claim 1, wherein said means for processing 

includes means for calculating color weighting factors. 

3 . The color wide dynamic range video imaging 
apparatus of Claim 2, wherein said means for processing 

25 includes means for calculating saturation color suppression 
factors . 

4 . The color wide dynamic range video imaging 
apparatus of Claim 1, wherein said means for dividing each 

3 0 color video image into components includes a filter means 

in front of said sensor means and said filter means 
includes filter elements of a plurality of colors, said 
plurality of colors corresponding to said components. 
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5. The color wide dynamic range video imaging 
apparatus of Claim 4, wherein said sensor means includes a 
plurality of pixel sensing elements and 

wherein said filter elements are arranged in a regular 
5 repeating pattern with each filter element in front of a 
single pixel element of said sensing means. 

6. The color wide dynamic range video imaging 
apparatus of Claim 5, wherein said processing means 

10 includes means for evaluating an intensity of each color 
component ; 

means for evaluating color suppression factors for 
each pixel; and 

means for converting said components into RGB color 
15 space. 

7 . The color wide dynamic range video imaging 
apparatus of Claim 6, wherein said means for evaluating an 
intensity of each color component communicates with a means 

2 0 for substituting luminance values when luminance otherwise 

approaches satur a t i on . 

8. The color wide dynamic range video imaging 
apparatus of Claim 6, wherein said means for evaluating an 

25 intensity of each color component communicates with a means 
for limiting luminance values. 

9 . The color wide dynamic range video imaging 
apparatus of Claim 5, wherein said processing means 

3 0 includes white balancing means which calculates correction 

factors for said components based upon the intensity of 
said color video images. 

10. The color wide dynamic range video imaging 
3 5 apparatus of Claim 9, wherein white balancing means 
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calculates an average intensity of said color video ? mages 
excluding saturated pixels and cut-off pixels* 

11. The color wide dynamic range video imaging 
5 apparatus of Claim 9, including vertical low-pass filter 

means receiving intensity and false edge suppression means, 
wherein said false edge suppression means receives 
intensity data from said vertical low-pass filter means and 
calculates false edge suppression factors which are reduced 
10 when said intensity data exceeds a pre-selected saturation 
value. 

12 . The color wide dynamic range video imaging 
apparatus of Claim 11, wherein said false edge suppression 

15 means calculates said false edge suppression factors for an 
image of a longest of said different exposure levels. 

13 . The color wide dynamic range video imaging 
apparatus of Claim 12, further including edge detection 

2 0 means and wherein an output of said edge detection means is 

multiplied times said false edge suppression factors. 

14. The color wide dynamic range video imaging 
apparatus of Claim 5, wherein color components are 

25 calculated by multiplying an intensity times a sum of 
products of prior color components and output of said means 
for calculating said color weighting factors, 

15. The color wide dynamic range video imaging 

3 0 apparatus of Claim 6, including means for calculating white 

balance correction factors for said components converted 
into RGB color space, wherein said white balance correction 
factors for a given respective RGB component are calculated 
by dividing average overall luminance by a multiple of an 
3 5 average value of said respective given RGB component. 
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16. The color wide dynamic range video imaging 
apparatus of Claim 15, wherein said average value of said 
respective given RGB component is calculated excluding 
pixels which are substantially near saturation or 

5 substantially near cut-off. 

17 . The color wide dynamic range video imaging 
apparatus of Claim 6, including means for converting said 
RGB color space components into mosaic color components 

10 corresponding to colors of said filter elements of said 
filter means. 

18 . The color wide dynamic range video imaging 
apparatus of Claim 1, wherein said means for processing is 

15 implemented on a single chip. 

19. An imaging apparatus comprising: 

sensor means for providing a plurality of color video 
images of a scene at different exposure levels; 

2 0 means for dividing each color video image into 

components ; and 

means for processing said components of each of said 
plurality of video images to produce a combined color video 
image including image information from said components of 
25 each of said plurality of color video images by applying 
neighborhood transforms to at least one of said components 
of each of said plurality of video images, 

wherein said means for processing includes means for 
calculating point intensity data for said each of said 

3 0 plurality of said video images, and 

wherein said processing means includes white balancing 
means which calculates correction factors for said 
components based upon the intensity of said color video 
images • 

35 
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20 • A color wide dynamic range video processing chip 
comprising: 

means for processing components of each of a plurality 
of video images of a scene at different exposure levels to 
5 produce a combined color video image including image 
information from said components of each of said plurality 
of color video images by applying neighborhood transforms 
to at least one of said components of each of said 
plurality of video images, 
10 wherein said means for processing includes means for 

calculating point intensity data for said each of said 
plurality of said video images. 
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